Implement RFC 1946 - intra-rustdoc links #47046

Manishearth · 2017-12-28T11:34:21Z

rust-lang/rfcs#1946 #43466

Note for reviewers: The plain line counts are a little inflated because of how the markdown link parsing was done. Read the file diff with "whitespace only" changes removed to get a better view of what actually changed there.

This pulls the name/path resolution mechanisms out of the compiler and runs it on the markdown in a crate's docs, so that links can be made to SomeStruct directly rather than finding the folder path to struct.SomeStruct.html. Check the src/test/rustdoc/intra-paths.rs test in this PR for a demo. The change was... a little invasive, but unlocks a really powerful mechanism for writing documentation that doesn't care about where an item was written to on the hard disk.

Items included:

Make work with the hoedown renderer
Handle relative paths
Parse out the "path ambiguities" qualifiers ([crate foo], [struct Foo], [foo()], [static FOO], [foo!], etc)
Resolve foreign macros
Resolve local macros
Handle the use of inner/outer attributes giving different resolution scopes (handling for non-modules pushed to different PR)

Items not included:

Make sure cross-crate inlining works (blocked on refactor described in Implement RFC 1946 - intra-rustdoc links #47046 (comment))
Implied Shortcut Reference Links (where just doing [::std::iter::Iterator][] without a reference anchor will resolve using the reference name rather than the link target) (requires modifying the markdown parser - blocked on Hoedown/Pulldown switch and Option to insert reference link definitions on demand pulldown-cmark/pulldown-cmark#121)
Handle enum variants and UFCS methods (Enum variants link to the enum page, associated methods don't link at all)
Emit more warnings/errors when things fail to resolve (linking to a value-namespaced item without a qualifier will emit an error, otherwise the link is just treated as a url, not a rust path)
Give better spans for resolution errors (currently the span for the first doc comment is used)
Check for inner doc comments on things that aren't modules

I'm making the PR, but it should be noted that most of the work was done by Misdreavus 😄

(Editor's note: This has become a lie, check that commit log, Manish did a ton of work after this PR was opened >_>)

rust-highfive · 2017-12-28T11:34:33Z

r? @GuillaumeGomez

(rust_highfive has picked a reviewer for you, use r? to override)

eddyb · 2017-12-28T13:21:33Z

src/librustdoc/visit_ast.rs

@@ -39,21 +39,21 @@ use doctree::*;
 // also, is there some reason that this doesn't use the 'visit'
 // framework from syntax?

-pub struct RustdocVisitor<'a, 'tcx: 'a> {
-    cstore: &'tcx CrateStore,
+pub struct RustdocVisitor<'a, 'b: 'a, 'tcx: 'b, 'rcx: 'b> {


Investigate whether 'b is actually needed.

eddyb · 2017-12-28T13:21:45Z

src/librustdoc/visit_lib.rs

@@ -22,8 +22,8 @@ use clean::{AttributesExt, NestedAttributesExt};

 /// Similar to `librustc_privacy::EmbargoVisitor`, but also takes
 /// specific rustdoc annotations into account (i.e. `doc(hidden)`)
-pub struct LibEmbargoVisitor<'a, 'b: 'a, 'tcx: 'b> {
-    cx: &'a ::core::DocContext<'b, 'tcx>,
+pub struct LibEmbargoVisitor<'a, 'b: 'a, 'tcx: 'b, 'rcx: 'b> {


Would be neat if we can remove 'b here.

eddyb · 2017-12-28T13:22:01Z

src/librustdoc/lib.rs

@@ -12,7 +12,7 @@
       html_favicon_url = "https://doc.rust-lang.org/favicon.ico",
       html_root_url = "https://doc.rust-lang.org/nightly/",
       html_playground_url = "https://play.rust-lang.org/")]
-#![deny(warnings)]


Need to put this back.

QuietMisdreavus · 2017-12-28T17:43:33Z

I got @eddyb's remarks, a couple nits of my own, and the tidy errors. I also updated it to work with Hoedown as well. I'd like to add a few tests before i'm ready to call it good.

QuietMisdreavus · 2017-12-28T23:26:52Z

This current travis failure: Somehow, one of the markdown changes has affected how item summaries are rendered. It looks like whatever happened to the Pulldown rendering made it strip out the "Read more" link on things like trait method descriptions. Investigating...

Manishearth · 2017-12-29T04:50:05Z

Implied Shortcut Reference Links (where just doing [::std::iter::Iterator][] without a reference anchor will resolve using the reference name rather than the link target)

Do we need to implement this this way? Bear in mind that [::std::iter::Iterator] desugars to [::std::iter::Iterator][::std::iter::Iterator] already so what we do need to do is handle that case. The empty-reference trick seems unnecessary.

edit: reread the spec, this is for both cases

Manishearth · 2017-12-29T06:36:27Z

My hunch was correct; fixed the "read more" bug.

Manishearth · 2017-12-29T07:02:28Z

Thinking about the shorthand more: Neither hoedown or pulldown let us intercept [foo] and [foo][bar] style markdown links with missing references; they turn them into plaintext too early. So we're in a bit of a pickle there. We could reparse; but ... ick.

(It is not hard to make this work if we're allowed to edit hoedown and pulldown. I'd rather not.)

Manishearth · 2017-12-29T09:13:49Z

Handled error recovery

Manishearth · 2017-12-29T09:16:57Z

One thing that's missing is that if you try resolving an enum variant it will link to the enum, and if you try resolving a method it won't work at all. Haven't looked into that.

href() doesn't support fragments, I think, so we'll have to make that work. Added to the TODO as an optional thing.

Manishearth · 2017-12-29T09:19:31Z

Btw, foreign items do not work. Also, because we generate the module page before its items, the cache isn't populated when we attempt to generate the markdown for the module page, and href fails. So the short summary stuff doesn't work either.

We probably need to hook directly into the stuff populating the cache for this to work. I'm not sure how easy that is.

QuietMisdreavus · 2017-12-29T16:04:50Z

@Manishearth What situation are you seeing where the cache isn't there? The test i added works. Is it when you're linking to stuff in submodules?

Unless i'm mistaken, the stuff in the cache that it's looking through should be populated in the previous stage of documentation. If it's not, it shouldn't be difficult to move it there.

Manishearth · 2017-12-29T16:31:06Z

The situation was when you're doing a link in the first line and you're looking at the short oneline summary docs on the module page. I don't know if it's a cache issue; I haven't investigated. I assumed it was a cache issue since the link resolves correctly on the item page and we don't do relative paths yet so it should be the exact same code running.

…

On Dec 29, 2017 9:35 PM, "QuietMisdreavus" ***@***.***> wrote: @Manishearth <https://github.com/manishearth> What situation are you seeing where the cache isn't there? The test i added works. Is it when you're linking to stuff in submodules? Unless i'm mistaken, the stuff in the cache that it's looking through should be populated in the previous stage of documentation. If it's not, it shouldn't be difficult to move it there. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#47046 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABivSFlDq1gWvtaWjDsTQuSmAYqRBduOks5tFQ2_gaJpZM4ROPwe> .

QuietMisdreavus · 2017-12-29T21:51:37Z

Is it the same issue as #38386? Summary lines on trait methods haven't had links ever, but i don't remember offhand if module item summaries do the same thing.

Manishearth · 2017-12-30T02:23:08Z

No so it shows up as a link, just that it's a link to ::foo, not fun.foo.html or whatever. I think links in mod have worked.

…

On Dec 30, 2017 3:22 AM, "QuietMisdreavus" ***@***.***> wrote: Is it the same issue as #38386 <#38386>? Summary lines on trait methods haven't had links ever, but i don't remember offhand if module item summaries do the same thing. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#47046 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABivSE9u1hTNTvo1M5ANUmNFrvLHZIOeks5tFV8CgaJpZM4ROPwe> .

Manishearth · 2018-01-01T06:26:33Z

Gah, it was a silly bug, we were pulling stuff off the wrong item. My bad.

Manishearth · 2018-01-01T08:36:24Z

@QuietMisdreavus happy new year Manishearth@7effbeb 😄 🎉

Manishearth · 2018-01-01T09:51:02Z

I don't have a satisfactory solution for cross-crate inlining. What we could do is shove a .links map somewhere in the target dir and reuse that, i guess.

QuietMisdreavus · 2018-01-02T14:50:40Z

The way we solved cross-crate inlining for #[doc(include="file.md")] was to do the processing in libsyntax and stuff the results back into another attribute so it would get saved in the metadata for the crate. However, to apply that procedure here we would need to integrate the markdown parser(s) into libsyntax, to properly extract all the links. I'm not exactly comfortable with making the Rust syntax parser also interpret Markdown as well, though. >_>

Manishearth · 2018-01-02T14:59:49Z

Yeah, that won't work well here. We could design a good way of persisting such info cross crate in the output docdir somewhere.

This does mean that such links will always be broken if you use the rustdoc tool directly.

@eddyb do you have any suggestions?

eddyb · 2018-01-02T17:35:00Z

What you actually want to persist cross-crate is the scope information that lets you resolve names in a certain (module) scope, after the fact. It'd also allow not doing all the the resolver hacks.

And there's a very good reason to want it in the compiler, at least same-crate: we could (finally) tell if a type is in scope somewhere so we can actually print Vec<_> instead of std::vec::Vec<_> etc.

So I think what you'll trying to do here is pretty cool, but maybe a bit ambitious at the moment?
And adding more hacks is delaying a really nice solution.

cc @jseyfried @petrochenkov

Manishearth · 2018-01-03T05:07:14Z

In that case perhaps we can land this without attempting to handle foreign items, and wait for a refactoring of resolution to do that.

QuietMisdreavus · 2018-01-05T16:29:45Z

I'm wary of landing this without cross-crate inlining, but if we can expedite the "really nice solution" that @eddyb mentioned, then it may be okay. There's enough here that's good, and it's not like the folder-path links are going away. (It just feels like an impotent feature without it, especially for something like std where it would help out the most. >_>)

So! Before we rush to get this reviewed, let's take stock of what we wanted to have here and what we want to push to later implementation. (Specifically that checklist in the OP that we keep editing.) I'm going to give macro resolution a try. I gave it an initial shot last week, but my initial implementation wasn't working, so it may need to be postponed as well.

@Manishearth What were you meaning with that one checklist item for "enums and UFCS methods"? Are enum variants and trait methods not available in the Resolver using ::?

QuietMisdreavus · 2018-01-05T19:15:28Z

I just ran the test manually to debug the macro resolution and noticed a bunch of "rendering difference" errors. When i ran it with Pulldown, i noticed something off.

Apparently, links with spaces in them don't work in commonmark at all.

In this crate we would like to link to:

* [`ThisType`](struct ::ThisType)
* [`ThisEnum`](enum ::ThisEnum)
* [`ThisTrait`](trait ::ThisTrait)
* [`ThisAlias`](type ::ThisAlias)
* [`ThisUnion`](union ::ThisUnion)
* [`this_function`](::this_function())
* [`THIS_CONST`](const ::THIS_CONST)
* [`THIS_STATIC`](static ::THIS_STATIC)

<p>In this crate we would like to link to:</p>
<ul>
<li>[<code>ThisType</code>](struct ::ThisType)</li>
<li>[<code>ThisEnum</code>](enum ::ThisEnum)</li>
<li>[<code>ThisTrait</code>](trait ::ThisTrait)</li>
<li>[<code>ThisAlias</code>](type ::ThisAlias)</li>
<li>[<code>ThisUnion</code>](union ::ThisUnion)</li>
<li><a href="::this_function()"><code>this_function</code></a></li>
<li>[<code>THIS_CONST</code>](const ::THIS_CONST)</li>
<li>[<code>THIS_STATIC</code>](static ::THIS_STATIC)</li>
</ul>

This completely renders the "path ambiguity" qualifiers useless because right now they'll generate a bunch of rendering warnings, and once we take out Hoedown they'll take out the links entirely. Our options are (1) taking them out, or (2) using something other than the space to separate the qualifier from the path. Something like # maybe?

eddyb · 2018-01-05T19:26:25Z

I've suggested unified pages before, such that you'd have doc.rust-lang.org/std/vec/Vec#struct.

bors · 2018-01-22T20:30:11Z

💔 Test failed - status-travis

QuietMisdreavus · 2018-01-22T21:01:46Z

[01:25:34] �[m�[m�[32m�[1m Documenting�[m rustc v0.0.0 (file:///checkout/src/librustc)
[01:26:04] error: unknown start of token: `
[01:26:04]  --> <stdin>:1:1
[01:26:04]   |
[01:26:04] 1 | `foo` `f` Call#1 `y` Yield `bar` `g` Call#3 Call#2 Call#0
[01:26:04]   | ^
[01:26:04]   |
[01:26:04] help: unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it's not
[01:26:04]  --> <stdin>:1:1
[01:26:04]   |
[01:26:04] 1 | `foo` `f` Call#1 `y` Yield `bar` `g` Call#3 Call#2 Call#0
[01:26:04]   | ^
[01:26:04] 
[01:26:05] �[m�[m�[31m�[1merror:�[m Could not document `rustc`.

I... have no idea where this even is. There's no file listed here. >_>

QuietMisdreavus · 2018-01-22T21:22:55Z

Found the culprit, but again, i'm not sure how this got here.

rust/src/librustc/middle/region.rs

Lines 349 to 356 in fdc18b3

    
           /// Obviously, the result of `f()` was created before the yield 
        
           /// (and therefore needs to be kept valid over the yield) while 
        
           /// the result of `g()` occurs after the yield (and therefore 
        
           /// doesn't). If we want to infer that, we can look at the 
        
           /// postorder traversal: 
        
           /// ``` 
        
           /// `foo` `f` Call#1 `y` Yield `bar` `g` Call#3 Call#2 Call#0 
        
           /// ```

I feel like this should have been caught in the main #46278 PR? o_O

ollie27 · 2018-01-22T21:29:17Z

Those docs are on a private field so prior to this PR they weren't rendered at all. Now they are "rendered" to look for links. The markdown_links function shouldn't be using hoedown_block or CodeBlocks anyway as they won't affect finding links.

QuietMisdreavus · 2018-01-22T23:07:39Z

Good catch! I just pushed a commit to bypass processing code blocks in markdown_links.

Manishearth · 2018-01-23T02:50:06Z

@bors r=eddyb,GuillaumeGomez,QuietMisdreavus,Manishearth

New commit lgtm

bors · 2018-01-23T02:50:07Z

📌 Commit 63811b6 has been approved by eddyb,GuillaumeGomez,QuietMisdreavus,Manishearth

bors · 2018-01-23T07:39:00Z

⌛ Testing commit 63811b6 with merge 48a7ea9...

…Gomez,QuietMisdreavus,Manishearth Implement RFC 1946 - intra-rustdoc links rust-lang/rfcs#1946 #43466 Note for reviewers: The plain line counts are a little inflated because of how the markdown link parsing was done. [Read the file diff with "whitespace only" changes removed](https://github.com/rust-lang/rust/pull/47046/files?w=1) to get a better view of what actually changed there. This pulls the name/path resolution mechanisms out of the compiler and runs it on the markdown in a crate's docs, so that links can be made to `SomeStruct` directly rather than finding the folder path to `struct.SomeStruct.html`. Check the `src/test/rustdoc/intra-paths.rs` test in this PR for a demo. The change was... a little invasive, but unlocks a really powerful mechanism for writing documentation that doesn't care about where an item was written to on the hard disk. Items included: - [x] Make work with the hoedown renderer - [x] Handle relative paths - [x] Parse out the "path ambiguities" qualifiers (`[crate foo]`, `[struct Foo]`, `[foo()]`, `[static FOO]`, `[foo!]`, etc) - [x] Resolve foreign macros - [x] Resolve local macros - [x] Handle the use of inner/outer attributes giving different resolution scopes (handling for non-modules pushed to different PR) Items not included: - [ ] Make sure cross-crate inlining works (blocked on refactor described in #47046 (comment)) - [ ] Implied Shortcut Reference Links (where just doing `[::std::iter::Iterator][]` without a reference anchor will resolve using the reference name rather than the link target) (requires modifying the markdown parser - blocked on Hoedown/Pulldown switch and pulldown-cmark/pulldown-cmark#121) - [ ] Handle enum variants and UFCS methods (Enum variants link to the enum page, associated methods don't link at all) - [ ] Emit more warnings/errors when things fail to resolve (linking to a value-namespaced item without a qualifier will emit an error, otherwise the link is just treated as a url, not a rust path) - [ ] Give better spans for resolution errors (currently the span for the first doc comment is used) - [ ] Check for inner doc comments on things that aren't modules I'm making the PR, but it should be noted that most of the work was done by Misdreavus 😄 (Editor's note: This has become a lie, check that commit log, Manish did a ton of work after this PR was opened `>_>`)

bors · 2018-01-23T10:21:51Z

☀️ Test successful - status-appveyor, status-travis
Approved by: eddyb,GuillaumeGomez,QuietMisdreavus,Manishearth
Pushing 48a7ea9 to master...

Manishearth · 2018-01-23T10:22:54Z

WOOOOOOOOOO

sgrif · 2018-01-23T14:35:40Z

So will this "just work" if I compile master and try it out? Is it feature gated somehow?

Manishearth · 2018-01-23T14:52:05Z

Yep, it should just work. It's nightly-gated; so if you try this with a stable compiler in 12 weeks stuff won't look great.

sgrif · 2018-01-23T18:26:59Z

👍 time to compile from master then. My office was getting a tad cold anyway.

sgrif · 2018-01-23T20:04:54Z

Hm is there additional magic I have to do for this? I tried changing a line that was

[`ToSql`]: ../../../serialize/trait.ToSql.html

to

[`ToSql`]: serialize::ToSql

but it just leaves serialize::ToSql as the href

KillTheMule · 2018-01-23T20:29:56Z

Can't check, but shouldn't there be brackets somewhere? Like [ToSql](serialize::ToSql)?

(e) No idea how to get the backticks correct, but I guess everyone can guess what I mean...

sgrif · 2018-01-23T20:33:45Z

Yes, the actual link in the text appears as [``ToSql``] (with only one backtick of course)

QuietMisdreavus · 2018-01-23T20:35:44Z

Is the serialize module in-scope at that definition site? It should only resolve the link if that path would resolve where that comment would be written.

sgrif · 2018-01-23T20:44:18Z

Gotcha. Changing it to ::serialize works as expected. I suppose it makes sense that it resolves as dispatch would and not use

@QuietMisdreavus

… r=QuietMisdreavus rustdoc: Fix link title rendering with hoedown The link title needs to be HTML escaped. It was broken by rust-lang#47046. r? @QuietMisdreavus

See: https://rust-lang.github.io/rfcs/1946-intra-rustdoc-links.html and: rust-lang/rust#47046 🎉

rust-highfive assigned GuillaumeGomez Dec 28, 2017

kennytm added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Dec 28, 2017

Manishearth mentioned this pull request Dec 28, 2017

Tracking issue (Rustdoc): hoedown -> pulldown migration #44229

Closed

10 tasks

eddyb reviewed Dec 28, 2017

View reviewed changes

QuietMisdreavus mentioned this pull request Dec 28, 2017

Tracking issue for RFC 1946 - intra-rustdoc links #43466

Closed

27 tasks

don't process code blocks when scanning for links

63811b6

bors merged commit 63811b6 into rust-lang:master Jan 23, 2018

Manishearth deleted the intra-doc-links branch January 23, 2018 10:22

bors mentioned this pull request Jan 23, 2018

WIP: Parallelize passes using rayon #46564

Closed

ollie27 mentioned this pull request Jan 29, 2018

rustdoc: Fix link title rendering with hoedown #47855

Merged

hobofan mentioned this pull request Feb 19, 2018

Add line numbers for dead links deadlinks/cargo-deadlinks#14

Open

rrbutani added a commit to ut-utp/core that referenced this pull request Oct 9, 2019

misc: fix up intra doc links (make use of rfc 1946)

6fc4e6b

See: https://rust-lang.github.io/rfcs/1946-intra-rustdoc-links.html and: rust-lang/rust#47046 🎉

Manishearth mentioned this pull request Oct 30, 2019

Support intra-doc links on cross-crate reexports #65983

Closed

jyn514 mentioned this pull request Dec 1, 2021

private macros break intra-doc links #81633

Closed

Implement RFC 1946 - intra-rustdoc links #47046

Implement RFC 1946 - intra-rustdoc links #47046

Conversation

Manishearth commented Dec 28, 2017 • edited by QuietMisdreavus Loading

rust-highfive commented Dec 28, 2017

eddyb Dec 28, 2017

Choose a reason for hiding this comment

eddyb Dec 28, 2017

Choose a reason for hiding this comment

eddyb Dec 28, 2017

Choose a reason for hiding this comment

QuietMisdreavus commented Dec 28, 2017

QuietMisdreavus commented Dec 28, 2017

Manishearth commented Dec 29, 2017 • edited Loading

Manishearth commented Dec 29, 2017

Manishearth commented Dec 29, 2017 • edited Loading

Manishearth commented Dec 29, 2017

Manishearth commented Dec 29, 2017

Manishearth commented Dec 29, 2017

QuietMisdreavus commented Dec 29, 2017

Manishearth commented Dec 29, 2017 via email

QuietMisdreavus commented Dec 29, 2017

Manishearth commented Dec 30, 2017 via email

Manishearth commented Jan 1, 2018

Manishearth commented Jan 1, 2018

Manishearth commented Jan 1, 2018

QuietMisdreavus commented Jan 2, 2018

Manishearth commented Jan 2, 2018

eddyb commented Jan 2, 2018 • edited Loading

Manishearth commented Jan 3, 2018 • edited Loading

QuietMisdreavus commented Jan 5, 2018

QuietMisdreavus commented Jan 5, 2018

eddyb commented Jan 5, 2018

bors commented Jan 22, 2018

QuietMisdreavus commented Jan 22, 2018

QuietMisdreavus commented Jan 22, 2018

ollie27 commented Jan 22, 2018

QuietMisdreavus commented Jan 22, 2018

Manishearth commented Jan 23, 2018

bors commented Jan 23, 2018

bors commented Jan 23, 2018

bors commented Jan 23, 2018

Manishearth commented Jan 23, 2018

sgrif commented Jan 23, 2018

Manishearth commented Jan 23, 2018

sgrif commented Jan 23, 2018

sgrif commented Jan 23, 2018

KillTheMule commented Jan 23, 2018 • edited Loading

sgrif commented Jan 23, 2018

QuietMisdreavus commented Jan 23, 2018

sgrif commented Jan 23, 2018

Manishearth commented Dec 28, 2017 •

edited by QuietMisdreavus

Loading

Manishearth commented Dec 29, 2017 •

edited

Loading

Manishearth commented Dec 29, 2017 •

edited

Loading

eddyb commented Jan 2, 2018 •

edited

Loading

Manishearth commented Jan 3, 2018 •

edited

Loading

KillTheMule commented Jan 23, 2018 •

edited

Loading